The present invention relates generally to optical networks, and more particularly, to multiple-in-multiple-out orthogonal frequency division multiplexing MIMO-OFDM based flexible rate intra-data center network.
The global Internet traffic is growing rapidly. Many of the Internet applications are hosted in the data center servers (such as cloud computing application, search engine, etc.) and are data-intensive and require large degree of interactions among the servers in the data center. (To distinguish from the communication network between different data centers (i.e. the “inter-data center network”), this network of interconnects among the servers in a data center is called the “intra-data center network” here. In some literatures, this is simply called “data center network”.) In the intra-data center network, the traffics among these servers are usually dynamic in nature and often busty. Therefore the intra-data center networks are facing challenging demand such as high channel bandwidth, flexible bandwidth configuration, low latency, high network throughput. Also, current data centers rely on electronic router to perform these interconnection functions, this consumes large amount of power and generates large volume of heat, which in turn requires more power for cooling. Therefore the intra-data center network also needs to have low energy consumption and low heat dissipation. An efficient intra-data center network architecture and corresponding interconnection scheme is required to meet these challenges.
Fiber optic technology is a natural way to solve these problems, because optical fiber offers very large bandwidth (10 G, 40 G, 100 G per channel or beyond), comparing to copper-based technology or wireless technology. Applying optical communication technology to the data center network also makes it easier to interface with inter-data center networks which have much larger traffic volume and therefore have been mainly based on optical communications technologies. The cost of optical transmission medium, namely the optical fiber, is also low cost, especially for multi-mode fiber. Many optical devices are passive and do not consume electricity. Based on these reasons, it is beneficial to apply optical technology in intra-data center network.
However, currently the optical technology in the intra-data center application is only for point-to-point links, which utilizes only the advantage of large bandwidth. Such network lacks flexibility. The switching between different servers is still relying on electronic switch fabric. And it does not offer the capability to share the bandwidth as the traffic pattern changes.
In the intra-data center network, the servers (usually up to 48 in the form of blades) are accommodated into racks and are connected through a Top-of-the Rack Switch (ToR). These ToR switches are further inter-connected through one or multiple tiers of aggregate switch in a fat-tree topology and switched at the core switch. Since the traffics within a rack is relatively smaller and the servers are physically located closer (thus easier to interconnect), the focus of the intra-data center network is on how to efficiently switch among the ToRs, since the ToR, the aggregate switch and the core switch in this architecture have high power consumption and cause latency in the traffic. This is the main target for this invention.
In the past few years, the optical communication research community has proposed several architectures to make the optical interconnects in the intra-data center network more efficient. Each of them has its advantages and disadvantages. Below are the descriptions of these prior arts and some analysis.
a. c-Through Architecture (FIG. 1)
The c-Through architecture was proposed by researchers from Rice University, Carnegie Mellon University and Intel. It improves the current electrically switched intra-data center network by using a hybrid electrical-optical architecture, where the ToR switches are connected to both an electrical packet-based network and an optical circuit-based network. An optical circuit switch is used connect pairs of rack with high bandwidth demands. It requires a traffic monitoring system to collect the bandwidth requirements and to determine the configuration of the optical switch. This architecture reduces completion time and offers significantly reduced latency for those applications in which the traffic demand between some hosts changes slowly.
This architecture requires a large sale core optical circuit switch, which is costly and present a large single point-of-failure. It also does not offer bandwidth sharing capability (unless it is done through electrical switch). Having both electrical switching system and optical switching system makes the network larger and more costly. The switching speed in this architecture is at the order of milliseconds (ms).
b. Helios Architecture (FIG. 2)
The Helios architecture was proposed by UCSD. It is also a hybrid electrical/optical switch architecture. It has a layer of electrical switches as the ToR switches (also called pod switches), then another layer of core switch, which include some electrical packet switches for all-to-all communication of the pod switches, and some optical circuit switched for traffic with high bandwidth yet changes slowly. It uses WDM (wavelength division multiplexing) transceivers to multiplex multiple high bandwidth channels to form superlinks to be switched optically.
This architecture also has the disadvantages of requiring large scale optical circuit switch and not able to share optical bandwidth. The switching speed is also at the ms order.
c. Proteus Architecture (FIG. 3)
This architecture was proposed by NEC and UIUC. It uses direct optical connections between ToR switches for high-volume connections and uses multi-hop connections (also optical) for low volume traffic. In this architecture, each ToR switch has multiple optical transceivers operating at different wavelengths, therefore can deliver coarse-grain flexible bandwidth by installing different number of transceivers. It then uses WSS (wavelength-selective switch) to send these wavelengths to different input port of an optical switch matrix for optical layer switching. This architecture needs some algorithm to find the optimum configuration for the core optical switch.
This architecture also requires a large core optical switch. The switch scale is even larger than the previous 2 architectures, because it requires multiple ports for each ToR. This increases the switch scale by several folds. The addition of WSS also increases the hardware cost. Using commercially available devices (WSS and core optical switch), the switching speed of this architecture is also at the ms range.
Recently a research paper published by Columbia University, Cornell University and BUPT proposed to use cascaded silicon microring switches and SOA (semiconductor optical amplifier) switch fabric to realize such architecture. This will increase the switching speed to the nanosecond (ns) range and thus capable for packet switching. However this remains at lab prototype level and not yet feasible in commercial deployment.
d. DOS Architecture (FIG. 4)
The DOS (Datacenter Optical Switch) architecture was proposed by UC Davis. It uses a cyclic arrayed waveguide grating (CAWG, sometimes it is called an AWG router or AWGR, sometimes it is also called a cyclic interleaver) that allows contention resolution in the wavelength domain. The CAWG is a N×N cyclic multiplexer/demultiplexer that routes different wavelengths from different inputs to different output ports in a cyclic manner, as illustrated in FIG. 5 (1 8×8 CAWG). In the DOS architecture, a tunable wavelength converter (TWC) is placed at every ToR switch to assign the appropriate wavelength to the signal so that it can be routed to the target output port through the CAWG and then received. In case that the number of output receivers is fewer than the input ToR switches, a SDRAM buffer is used to store the transmitted packets temporarily. Since each of the CAWG output can receive signals from all inputs, this buffer can be shared among all input ToR switches.
An advantage of this architecture is that the latency is almost independent of the number of input ports and remains low even at high input loads (if signal is not buffered), and it does not require a large scale core fiber switch. A disadvantage is that it requires electrical buffer for congestion management. This OEO (optical-electrical-optical) scheme is power hungry and causes latency. Also, it uses TWC for wavelength tuning, which is quite costly. The switching speed depends on the performance of the TWC, which ranges from several ns (lab prototype) to ms level (commercial products).
Several other architectures also use CAWG and wavelength converter to form the optical core. The first one is the Petabit switch fabric proposed by Polytechnic Institute of NY. In this architecture, a three-stage Clos network is used and each stage consists of an array of CAWGs, and an array of tunable lasers and two arrays of TWCs are used for wavelength tuning (i.e. route switching). This architecture does not require electrical buffers inside the switch fabric, but instead use electronic buffers in the line cards and an efficient scheduling algorithm to perform congestion management. The disadvantages include the requirement of large quantities of TWCs, tunable lasers and CAWGs, which is very costly.
Another architecture that use CAWG and wavelength converter was proposed by Alcatel-Lucent in an IRIS Project. It is also based on a three-stage switch to provide dynamically non-blocking switching. This architecture also requires multiple CAWGs and multiple arrays of wavelength converters, which is very costly. Besides that, it require arrays of wavelength switches and different optical delay lines for time buffer, which adds the hardware cost even further. Also, this architecture does not provide sharing of bandwidth between ToRs.
e. Optical OFDM-Based Data Center Network (FIG. 6)
The feasibility of using optical OFDM (orthogonal frequency division multiplexing) technology for data center application has been considered, but no actual network architecture is proposed for the intra-data center network in this paper.
An optical OFDM-based data center network architecture was proposed by Athens Information Technology. This architecture has the similar architecture of a ROADM (reconfigurable optical add/drop multiplexer) node based on WSS and combiner. Optical OFDM (where the OFDM subcarriers are generated optically) transceivers are used to connect the ToR to the centralized WSS-based switch, where the subcarriers are routed to different ports based on the traffic requirements. This enables relatively fine level bandwidth sharing at each ToR (the reason why we say it's “relatively” fine level sharing is that the sharing is done at the optical subcarrier level, which typically has the resolution of 10 Gb/s or higher). At the output port of the switch, all of the subcarriers are multiplexed and forwarded to the OFDM receiver which retrieves the subcarriers and forwards the packets to the servers. An efficient bandwidth allocation algorithm is needed to take advantage of the flexibility and finer grain tunability of OFDM.
The disadvantages of this architecture include: (1) It requires a flexible grid WSS for each ToR, which is not commonly available, has limited granularity, has limited port count (commercial prototypes are up to 1×9, therefore cannot support the data center network which has typically 100s or 1000s of ToRs), and costly. (2) It requires guard bands between subcarrier bands for different output ports, which leads to less efficient spectrum utilization. (3) It requires multiple electrical-optical modulators at each ToR, because typically the optical OFDM requires one modulator for each OFDM subcarrier. (4) The granularity is still not fine enough for efficient flexible bandwidth sharing. (5) It requires either high port count optical coupler (large optical loss) or WSS (costly) at each receiver to multiplex the switched signals.
Accordingly, there is a need for a flexible rate intra-data center network that overcomes the deficiencies of prior efforts.